Stereoscopic video encoding and decoding methods and apparatus

ABSTRACT

Methods and apparatus for stereoscopic image encoding and decoding are described. Left and right eye images are encoded following an entropy reduction operation being applied to one of the eye images when there is a difference between the left and right images of an image pair. Information about regions of negative parallax within the entropy reduced image of an image pair is encoded along with the images. Upon decoding a sharpening filter is applied to the image in an image pair which was subjected to the entropy reduction operation. In addition edge enhancement filtering is performed on the regions of the recovered entropy reduced image which are identified in the encoded image data as regions of negative parallax. Interleaving of left and right eye images at the input of the encoder combined with entropy reduction allows for efficient encoding, storage, and transmission of 3D images.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/107,852 filed Dec. 16, 2013 which is a divisional of U.S.patent application Ser. No. 12/862,457 filed Aug. 24, 2010 which claimsthe benefit of U.S. Provisional Patent Application Ser. No. 61/236,208filed Aug. 24, 2009. U.S. patent application Ser. No. 14/107,852 andU.S. patent application Ser. No. 12/862,457 are hereby expresslyincorporated by reference in their entirety.

FIELD

The present invention relates to the field of stereoscopic imagery andmore particularly, to the field of digital stereoscopic videoacquisition, distribution and playback including, e.g., methods andapparatus for encoding and decoding stereoscopic video and/or modifyingstereoscopic video for presentation.

BACKGROUND

Conventional stereoscopic video normally requires twice the storagecapacity, transmission bandwidth and playback bandwidth of 2-dimensionalvideo. This is because stereoscopic video requires that two imagestreams be produced, e.g., one for the left eye and one for the righteye. The burden of the second video data stream can overwhelm existingmodalities for content distribution (e.g. DVD disc, internettransmission, cable broadcast). Additionally, the burden of the secondvideo data stream can exceed the data transfer capabilities of localstorage media and the rendering capabilities of consumer digital videoplayback hardware.

In view of the above, it should be appreciated that there is a need forimproved methods of encoding stereoscopic video, decoding stereoscopevideo and/or manipulating stereoscopic video for presentation to takeinto consideration such factors as screen size and/or viewerpreferences.

SUMMARY

Various features and embodiments are directed to methods and apparatusfor encoding and/or decoding digital stereoscopic images, e.g., imagesforming a video.

In accordance with one embodiment of the present invention left andright eye images are interleaved and encoded.

Prior to encoding a pair of left and right eye images, the luminancecomponents of the left and right images may be analyzed to determine anamount of difference between the left and right eye images. Thedetermined amount of difference in some embodiments is indicated in adifference metric generated by comparing the left and right eye images.

In some embodiments if there is no difference between the luminancecomponents of the left and right eye images, the left and right eyeimages are encoded without being modified, e.g., using an interframecoding process. In some such embodiments, the coding involves includingin the encoded bitstream an indicator indicating that there is nodifference between the left and right eye images of an image pairallowing for efficient coding and avoiding the need to include a fullset of image data corresponding to both the left and right eye images.

If the difference determination indicates a difference between the leftand right eye images, in some embodiments one of the left and right eyeimages, e.g., the right eye image, is subject to an entropy reductionfiltering operation. In some embodiments the amount of entropy reductionfiltering applied is a function of the determined difference between theleft and right eye images. While application of entropy reduction to theright eye image is used in the example to explain the invention, theentropy reduction could be applied to the left eye image instead.

In some embodiments the amount of entropy reduction applied to the righteye image is greater when the difference between the left and right eyeimages is small and the amount of entropy reduction decreases as theamount of entropy increases. The entropy reduction filtering operation,in various embodiments, reduces the contrast within the image, e.g.,smoothes the image. This will, in many cases, increase coding efficiencyand reduce the amount of data required to represent the entropy reducedimage.

While the entropy reduction filtering applied to one of the images canincrease coding efficiency and thereby reduce the amount of data need torepresent a pair of left and right eye images, it has the potential toreduce the sharp edges of objects which are to be shown extending outfrom the screen. Visual cues provided by sharp edges in combination withthe amount of difference between the left and right eye images can beimportant for a good 3D image viewing experience.

In some but not necessarily all embodiments, the left and right eyeimages are analyzed as part of the encoding process to identify imageregions of negative parallax. In some embodiments informationidentifying the image regions in the entropy reduced image wherenegative parallax occurs is stored and included in the set of datarepresenting the encoded pair of left and right eye images. The regionidentifying information may be included with the encoded image data asmetadata along with the difference metric determined by comparing theleft and right eye images being encoded.

The encoding process may be repeated for multiple left and right eyeimage pairs with the encoded data forming a sequence of encoded left andright eye image pairs. The coding of the left eye image and entropyreduced right eye images is performed, in some embodiments, by supplyinga sequence of processed image pairs to an encoder such as an H.264 AVCcoder which performs encoding using motion compensated discrete cosinetransform based compression on the images, e.g., the interleaved leftand right eye frames. The result, e.g., sequence of encoded frames, ofthe encoding along with the metadata corresponding to encoded framepairs is stored in a file and/or transmitted.

While in some embodiments the encoded images are decoded and displayedallowing an operator of the encoding system to monitor the quality ofthe encoded images, the encoded bitstream is normally communicated to aplayback device for retrieval, decoding and display.

Various features are directed to decoding and displaying stereoscopicimages encoded in accordance with one or more of the encodingembodiments described herein.

In at least one decoding embodiment the encoded frames are decoded alongwith the metadata including the determined difference metric andindicating the regions of negative parallax in entropy reduced images.The decoding may be performed using a decoder which performs decoding ina manner consistent with the encoding/compression standard which wasused to encode the frames. The decoded left and right frames of eachframe pair are subject to further processing prior to display.

In some embodiments, a determination is made as to whether or notentropy reduction was performed on an image in the frame pair, e.g., theright eye image for purposes of explaining the invention. In oneembodiment this determination can be made by examining the differencemetric included in the metadata communicated with a frame pair. In someembodiments a zero difference metric indicates that entropy reductionwas not used while a non-zero metric indicates that entropy reductionwas performed on one of the images of a frame pair.

In one embodiment if one of the images in the decoded frame pair wassubject to entropy reduction filtering as part of the encoding process,a sharpen filtering operation is applied to the entropy reduced image.Portions of the sharpened image are then subject to edge enhancement. Insome embodiments the edge enhancement operation is applied to the imageregions identified as being regions of negative parallax regions, e.g.,as indicated by the metadata. In some embodiment the regions to befiltered are determined from polygonal region information associatedwith the frame pair being processed. This information may be recoveredfrom the metadata associated with the frame being processed.

The edge enhancement operation enhances, e.g., strengths, edges withinthe image region subject to the edge enhancement. Thus, edge in imageregions where negative parallax was detected in the entropy reducedimage being processed are enhanced. This helps restore the sharp edgesof three dimensional images which facilitates, along with the differencebetween the left and right eye images, the perception by a viewer of the3D nature of an image. The edge enhancement is limited, in some but notnecessarily all embodiments, to image portions where negative parallaxwas detected to avoid strengthening of edges due to block codingartifacts throughout the image. Thus, while edge sharpening may bedesirable for 3D regions in areas of negative parallax, applying it tothe entire image may be undesirable and is avoided for this reason insome embodiments.

In some embodiments, the strength of the edge enhancement filtering is afunction of the difference metric communicated with the image pair beingprocessed. The lower the difference metric the stronger the edgeenhancement. This counters the relatively higher amount of entropyreduction filtering applied to images with lower difference metrics thanis applied to images with high difference metrics.

The decoded full resolution image and processed entropy reduced image ofan image pair, in the case of an image pair where entropy reduction wasapplied to one of the images, is supplied to a processor which rendersthe images for output to a 3D display device in accordance with therendering requirements of the particular display.

In cases where entropy reduction was not applied to either of the leftand right eye images since there was no detected difference between theluminance components of the images, the decoded left and right eyeimages are supplied to the processor for rendering.

Thus, images which were subjected to entropy reduction as part of theencoding process are enhanced to reduce the potential effects of theentropy reduction while images which were not subject to entropyreduction are decoded and provided for rendering without the use or needfor the enhancement operations performed on the entropy reduced images.

The encoding and decoding methods of the present invention which involvethe interleaving of left and right eye images as part of an interframeencoding process, take advantage of the high degree of similaritybetween left and right eye images. Since many eye images may not include3D effects or features, a large number of image pairs will include twoeye images which are the same allowing for one image to be efficientlycoded by simply indicating a no difference or repeat of the other eyeimage in the left and right eye image pair. For image pairs where thereis a difference, the selective application of entropy reduction with theamount of entropy filtering depending on the difference between imagesin the image pair, results in an efficient encoding method due toreduction in image entropy while still providing good image results and3D image effects.

While various embodiments have been discussed in the summary above, itshould be appreciated that not necessarily all embodiments include thesame features and some of the features described above are not necessarybut can be desirable in some embodiments. Numerous additional features,embodiments and benefits of various embodiments are discussed in thedetailed description which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1-13 illustrate various processing steps which may, and sometimesare, performed in one exemplary embodiment.

FIG. 14 illustrates a computer system or other hardware encoder and/orplayer which may be used to perform video encoding, storage,transmission, reception decoding, display and/or other operations inaccordance with the invention.

DETAILED DESCRIPTION

The present invention is directed to methods and apparatus for encodingand decoding stereoscopic video and/or modifying stereoscopic video forpresentation.

FIG. 1 illustrates steps 1 of processing implemented in accordance withone exemplary embodiment of the invention. Step 1 100 involvesperforming stereo video image acquisition and editing. As shown in box102, in step 100 a stereo 3D scene 104 is captured by separate left andright cameras 106, 108 which produce left and right images 110, 112. Theleft and right eye images may be represented by pixel values. In amonochrome embodiment the pixel values are simply luminance pixelvalues. Step 100 may be implemented using known image acquiring andediting methods. Operation proceeds from step 100 to step 2A 200.

Step 2A 200 includes comparing the right eye image 112 and the left eyeimage 110 luminance information to identify areas of difference andgenerate a difference metric as a function of the amount of detecteddifference between the left and right eye images 110, 112. Thedifference metric is stored, e.g., in memory, for subsequent use andinclusion as metadata with the pair of eye images being encoded. In oneembodiment the luminance value of a pixel in the left eye image 110 at alocation is XORd, as represented by XOR operation 208, with theluminance value of the pixel at the corresponding image location in theright eye image 112. If the two pixel values are the same the XORresults in a 0. However, if they are different a 1 is output. By XOR′ingthe pixel values of the left and right image conveying luminanceinformation, and summing the result, a count of the number of pixelswhich differ in the left and right eye images 110, 112 is generated.This count represents a difference metric 210 which is used to controlsubsequent processing in some embodiments. The larger the value of thedifference metric 210 the greater the indicated difference between theleft and right eye images 110, 112 of an image pair. In otherembodiments comparison techniques other than an XOR are used to detectdifferences between the left and right eye images 110, 112.

In addition to generating a difference metric, in step 2B 212 shown inFIG. 2, the left and right eye images 110, 112 are processed todetermine regions of one of the eye images, e.g., the right eye image112 for purposes of this example, of negative parallax. FIG. 2illustrates step 2B of processing implemented in accordance with oneexemplary embodiment of the invention. As shown in box 214, step 2Bincludes determining in step 216 regions of negative parallax. Theunshaded portions of image 218 represent detected regions of negativeparallax with the right eye image 112.

Operation proceeds from step 2B to step 3 300 shown in FIG. 3. Step 3300 includes determining if the difference metric (e.g., generated instep 2A) indicates a zero difference in luminance information betweenthe left and right eye images. As shown in box 302, sub-step 304represents a decision step the outcome of which controls how theprocessing proceeds based on the determination of whether the differencemetric indicated a zero difference or not. In decision step 304 if it isdetermined that the difference metric indicated a non-zero difference,e.g., a difference between the left and right eye images was detected,processing proceeds to step 4A as indicated in connecting node 306. Ifit is determined that the difference metric indicated a zero difference,the processing proceeds to step 5 500 as indicated in connecting node408. Thus, for images where there is no difference, the identificationof regions of negative parallax and entropy reduction operations areskipped reducing computational requirements as compared to if thesesteps were performed.

Operation proceeds from step 3 to step 4A 400 also shown in FIG. 3, whenit is determined that the difference metric indicated a non-zerodifference. Step 4A 400 includes generating information identifying thedetected regions of negative parallax in the right eye image and storingthe region identifying information. The stored information 408identifying regions of negative parallax in the right eye image mayindicate the perimeter of the parallax regions, e.g., as combination ofmathematical formula(s) and/or perimeter points. The defined polygonalregion(s) include the edges of the negative parallax regions as shown inbox 406 wherein the negative parallax regions are included in thedefined region perimeter shown in box 406. In various embodiments theinformation identifying regions of negative parallax includesinformation defining the perimeter of at least one polygonal region,e.g., as shown in box 406.

Operation proceeds from step 4A to step 4B. FIG. 4 illustrates step 4B410 which includes applying an entropy reduction filtering operation toright eye image 414 to reduce entropy. The filtering operating 416 maybe an FFT based filtering operation or a neighborhood filteringoperation. The entropy reduction filtering operation 416 reduces thecontrast within the image 112 and produces an entropy reduced right eyeimage 418. Box 412 in FIG. 4 shows subjecting the right eye image 112 toentropy reduction filtering operation 416 to generate the entropyreduced right eye image 418.

In various embodiments the amount of entropy reduction performed byfiltering step 416 is a function of the difference metric, e.g., thegreater the amount of difference, the lesser the smoothing. Thus ahigher difference metric, in some embodiments results in less entropyreduction than a lower difference metric.

FIG. 5 illustrates step 5 500, which is performed in accordance with oneexemplary embodiment. Step 5 500 includes combining left and right eyeimages of an image pair into a set of data representing a pair of framesin accordance with the invention. Step 5 500 also includes, in someembodiments, inserting information indicating regions of negativeparallax, e.g., polygonal regions formula, into data corresponding toright eye image, e.g., as frame metadata, to facilitate imageenhancement during decoding. In some embodiments the included metadataalso includes the difference metric (DM) 210 for the image pair whichwas determined in step 200. If entropy reduction was performed, step 500includes the entropy reduced right eye image otherwise the right eyeimage 112 is included in the image pair. Thus the pair of frames 510includes the left-eye view 110, the right-eye view 418 or 112 (theentropy-reduced eye image 418 is include when the difference metricindicates a non-zero difference indicating it was generated otherwisethe right eye image 112 is used) and information identifying regions ofnegative parallax in the right eye image 508 if it was generated.

Operation proceeds from step 5 to step 6 600 shown in FIG. 6. Step 6 600includes repeating steps 1 through 5, e.g., for left and right eye imagepairs in a video sequence being encoded, in accordance with theinvention. Thus as shown in box 602 frame pair 1 604 through frame pairn 606 are generated for a given video sequence being encoded.

The input to step 7 700 shown in FIG. 7 is the image data generated inFIG. 6. Note that the left and right eye image data is arranged so thatthe left and right eye images are presented for encoding/compression inan interleaved manner facilitating data compression and efficientencoding. Step 7 700 includes the compression of left eye image data andentropy right eye image data included in the pairs of frame data 604,606 to compressed frame data, i.e., compressed left and right eye imagedata and associated metadata. The encoded frame pairs generated bycompression step 708 form a video stream. Thus step 7 700 includes thecompression of the processed stereoscopic video sequence, in the form offrame pairs (e.g., frame pair 1 604, . . . , frame pairs n 606) by theelement 708, in accordance with one aspect of the invention, e.g., usingmotion compression algorithms which can, and in some embodiments are,implemented using one or more compression methods that may employ adiscrete cosine transform, Advanced video coding standard H.264 AVC isone example of a compression method that may be used to compress thesequence of frames in the frame pairs 604, 606 but other compressionstandards might be used instead. The output of the compressionoperation, i.e., the compressed frame pairs and associated metadata areindicated by reference 710.

FIG. 8 illustrates step 8 800 and step 9 900 which are performed in someembodiments. As should be appreciated playback may occur at a differentlocation from the location where the encoded video is produced, storedand/or transmitted from. However, in some embodiments to allow a user ofthe encoder to view and consider the quality of the encoded images,decoding and display of images may, and in some embodiments is,performed on the same system used to encode image data. Step 8 800includes the storage 804 and/or transmission 806 of the processed, e.g.,encoded, frame pairs generated in accordance with the invention. Theencoded information which is stored or transmitted includes the metadataindicating the difference metric and regions of negative parallax forentropy reduced images in frame pairs. In step 9 900 the processed framepair(s) generated in accordance with the invention is received by aplayback device, e.g., a player located at a customer premise or roomwhich may be remote from the storage and/or transmitter location, or issimply accessed from where it is stored. The playback device includes aprocessor, e.g., a graphics processing unit (GPU) capable of operatingas a video decoder in accordance with the invention and is capable ofrendering stereoscopic images from the encoded video received by theplayback device. The access and/or display operation is shown usingreference number 904.

FIG. 9 illustrates step 10 1000 showing decompression of encoded framesas part of the decoding and display process in accordance with anexemplary embodiment of the invention. Step 10 1000 includesdecompressing the received and/or retrieved first frame pair of theencoded frame pairs 710, to recover left eye image data and right eyeimage data and to recover metadata associated with the first frame pair1008. The processing is illustrated in the sequential manner as shown inbox 1002. The decompression operation is illustrated by element 1006 andthe recovered first frame pair 1008 including the left eye image data,right eye image data and the metadata is also shown. The metadataincludes, e.g., polygon region information corresponding to regions ofnegative parallax in the entropy reduced image of the decoded frame pairand the difference metric for the decoded frame pair determined duringthe encoding process. The decompression step 1006 may involve use of thedecoder or decoder portion of a codec used to the compression standardused to compress the frames being decoded. The decoded left frame 1012and decoded right frame 1014 are supplied in step 1010 to the processorfor additional processing. The right frame, e.g., right eye image 1014is shown as an entropy reduced right eye image but would be anon-entropy reduced image if entropy processing was not performed aspart of the encoding process, e.g., when there is zero differencebetween the left and right eye images.

Operation proceeds from step 10 to step 11A shown in FIG. 10. Step 11A1100 includes determining if entropy reduction was performed on one ofthe left and right eye images. In various embodiments the determinationwhether entropy reduction was performed on one of the left and right eyeimages, is performed by determining if there is a non-zero differenceindicated by the difference metric, e.g., included in the recoveredmetadata. Alternatively, in embodiments where the difference metric isnot communicated, the decoded left and right eye images may be comparedand a difference metric generated in the decoder.

As shown in box 1102, sub-step 1104 represents a decision step theoutcome of which controls how the processing may proceed based on thedetermination of step 11A, i.e., if entropy reduction was performed onone of the left and right eye images. In decision step 1104 if it isdetermined that entropy reduction was performed on one of the left andright eye images, the processing proceeds to step 11B as indicated inconnecting node 1106. If it is determined that entropy reduction was notperformed on one of the left and right eye images, the processingproceeds directly to step 12 as indicated in connecting node 1108.

In step 11B a sharpening filtering operation is performed on the imagewhich was subject to an entropy reduction operation, e.g., the right eyeimage. The sharpening tends to increase visual edge prominence withinright eye image. Thus a sharpened version of the entropy reduced righteye image is produced as a result of applying the sharpening filteroperation. The sharpen entropy reduced image 1106 is processed in step11C 1100.

FIG. 11 illustrates step 11C which includes the creation of a filtermask using the polygonal region information 1107, and application ofedge enhancement to image regions within the sharpened entropy reducedimage 1106 (the right eye image in the example) defined by the filtermask created using the polygonal region information 1107. Thus as shownin box 1102, in step 11C edge enhancement of portions of the decodedentropy reduced right eye image 1106, which are areas in which negativeparallax was detected, is performed by applying edge enhancementoperation. This edge enhancement strengths the edges within theidentified regions and tends to compensate for softened image edges dueto the entropy reduction process that might destructively interfere withstereoscopic depth cues. Thus edges which correspond to areas wherestereoscopic depth cues are of concern are enhanced more than edges inother portions of the image. The output of the edge enhancementoperation as done in step 11C is shown as sharpened edge enhanced righteye image 1110. Note that edge enhancement is not performed in someembodiments to the entire decoded entropy reduced image to avoidincreasing the strength of block coding artifacts throughout the entireimage, e.g., artificial edges at block coding boundaries resulting fromthe block coding used to generate the encoded frames.

With the recovered image data having been enhanced to improve 3D effectswhich may have been adversely impacted by the compression and entropyreduction process, operation proceeds to step 1220 which is a renderingstep. FIG. 12 illustrates step 12 1200 which includes the use of aprocessor, e.g., a graphic processing unit (GPU), to render the left-eyeview 1206 with the right-eye view 1208. The right eye view will be thesharpened edge enhanced right eye view for frame pairs where entropyreduction was previously applied to the right eye view. If no entropyreduction was applied to either eye image, the decoded frames aresupplied to the processor for rendering without enhancement. Therendered image(s) generated by the processor in step 1204 from therecovered left and right eye image data is output to a 3D display devicein step 1210. The rendering and output by the processor may be displaydevice dependent. In some but not necessarily all embodiments, acheckerboard mask 1208 is used in the image rendering process performedin step 1204. The rendering may be performed in a known manner using theleft and right image data 1206, 1208.

FIG. 13 illustrates step 13. In step 13 steps 10 to 12, as applicable,are repeated as the remaining frame pairs in the stereoscopic videosequence are decoded and output for display.

FIG. 14 illustrates a computer system 1400 which may be used to performvideo encoding, storage, transmission, reception, decoding, displayand/or other operations in accordance with the invention. It should beappreciated that while the system 1400 can be used to perform encodingand display options, more limited devices may perform some but not thefull set of functions supported by the system 1400. For example, aplayback device may implement the reception, decoding and displayfeatures but not the image encoding or capture features of the exemplaryembodiment.

The computer system 1400 includes a display device 1402, input device1404, I/O interface 1406, processor 1409, network interface 1410, andmemory 1412. The display device 1402 may be used, e.g., to displayimages resulting from processing implemented in accordance with thepresent invention. Input device 1404 may be, e.g. a keyboard or otheruser input device. The display 1402 and input device 1404 are coupled toa bus 1408 by I/O interface 1406. The bus 1408 is also coupled to thememory 1412, processor 1409 and network interface 1410. The networkinterface 1410 couples the internal components of the system 1400 to anexternal network, e.g., the Internet, thereby allowing the system 1400to receive and send data over a network. The processor 1409 controlsoperation of the computer system 1400 under direction of softwaremodules and/or routines stored in the memory 1412.

The I/O interface 1404 is used for receiving and/or transmittinginformation from other devices. For example, in some embodiments the I/Ointerface 1404 is used for transmitting the compressed left and righteye images to another device, e.g., a playback device. The I/O interface1404 is also used in some embodiments for receiving a file including theencoded frames of stereoscopic image data, said file including aplurality of frame pairs which were interleaved to form a left and righteye image sequence prior to encoding.

The memory 1412 includes a software control routines 1414 e.g., machineexecutable instructions, for implementing one or more of theabove-described processing methods of the present invention. Memory 1412includes the acquired left and right eye images image 1415, a differenceamount determination module 1416, a decision module 1418, a compressionmodule 1420 including an encoder module 1422, an entropy reductionfiltering module 1424, a negative parallax detection module 1429, adecoder module 1430, an entropy reduction determination module 1432, asharpening filter module 1436, a sharpening filter control module 1438,an edge enhancement operation module 1440, a rendering module 1442,stored information 1444, encoded left and right eye images file 1446,decoded left and right eye images file 1448, and rendered image 1450. Invarious embodiments the entropy reduction filtering module 1424 includesa FFT (Fast Fourier Transform) based filtering module 1426 andneighborhood filtering module 1428. In various embodiments the entropyreduction determination module 1432 includes a difference determinationmodule 1434.

Various modules shown in the memory 1412 are executed by the processor1409 to perform various functions in accordance with a method of thepresent invention. The acquired left and right eye images 1415 are theimages captured by separate left and right cameras. The acquired leftand right eye images 1415 may have undergone some processing, e.g., toput the acquired images in a suitable format for storing and further rprocessing. The left and right eye images form a first image pair. Thedifference amount determination module 1416 performing a comparisonoperation for comparing right eye image and left eye image to determinea difference between right eye image and left eye image of the firstimage pair. In various embodiments the difference amount determinationmodule 1416 compares right eye image and left eye image luminanceinformation to identify areas of difference, and produces a differencemetric as the output of comparison. In various embodiments thedifference metric 1445 indicating the amount of difference in theluminance information of the left and right eye images is stored in thememory 1412.

The decision module 1418 makes a decision whether or not an entropyreduction filtering operation is to be performed based on the amount ofdifference determined by the difference amount determination module1416, e.g., as indicated by the difference metric. In variousembodiments when it is determined that there is a non-zero amount ofdifference between the left and right eye images, the decision module1418 decides that entropy reduction filtering will be performed, andotherwise not. The entropy reduction filtering module 1420 performs anentropy reduction operation on right eye image data representing theright eye image of the first image pair to produce entropy reduced righteye image data, when it is determined that there is a non-zero amount ofdifference between the left and right eye images. In some embodimentsthe entropy reduction operation is a function of the determined amountof difference between said left eye image and the right eye image, theamount of entropy reduction being greater when the amount of differenceis a first amount than when it is a second amount, said second amountbeing greater than said first amount. Thus entropy reduction is greaterwhen the amount of difference is less. In various embodiments theentropy reduction filtering module 1420 includes a module 1422 and amodule 1424 for performing at least one of an FFT based filteringoperation or performing neighborhood filtering operation on said righteye image data to smooth the right eye image.

The compression module 1426 performs compression operation on the leftand right eye images in accordance with the invention, and includes anencoder module 1428 which performs encoding operation on the left eyeimage data representing the left eye image of the first image pair togenerate compressed left eye image data, and performs encoding on theentropy reduced right eye image data to generate compressed right eyeimage data, e.g., when entropy reduction has been performed on the righteye image. In various embodiments the encoder module 1428 performsinter-frame motion compensated encoding, data corresponding to said leftand right eye images being processed as data corresponding to sequentialframes. In some embodiments motion compensated discrete cosine transformbased compression, such as H.264 AVC is used for encoding. The file 1446including the compressed/encoded left and right eye images is an outputof the compression module 1426.

The negative parallax detection module 1429 detects image regions in theright eye image which are in negative parallax relative to acorresponding portion of the left eye image. In various embodimentsinformation identifying one or more detected image regions of negativeparallax in the right eye are stored in a file with the compressed leftand right eye image data, in the memory 1412. In some embodiments thenegative parallax determination 216 generates information identifyingregions of negative parallax in the right eye image and storeinformation, e.g., indicating perimeter of the parallax regions, e.g.,as a mathematical formula and perimeter points, in the memory 1412. Invarious embodiments the information identifying regions of negativeparallax includes information defining the perimeter of at least onepolygonal region, e.g., representing a detected region of negativeparallax. In some embodiments the file 1446 also includes informationidentifying at least one detected image region of negative parallax inthe right eye image.

The decoder module 1430 performs decompression and decoding on theencoded frames representing left and right eye images of an image pairto recover decoded left and right eye image data as part of the decodingand display process in accordance with an exemplary embodiment of theinvention. In various embodiments the metadata associated with the imagepair being decoded is also recovered during the decoding operationperformed by the decoder module 1430. The metadata includes, e.g.,polygon region information and in some embodiments the differencemetric. The file 1448 including the decoded left and right eye images isan output of the decoder module 1430.

The entropy reduction determination module 1432 determines whether ornot the entropy reduction operation was performed on one of the left andright eye images prior to encoding. In various embodiments thedetermination whether entropy reduction was performed on one of the leftand right eye images, is performed by determining if there is a non-zerodifference between said left and right eye images. In variousembodiments the entropy reduction determination module 1432 includes adifference determination module 1434 which determines the differencebetween decoded (recovered) left and right eye images. In someembodiments the entropy reduction determination module 1432 isconfigured to determine if a difference indicator value associated withat least one of the encoded left and right eye images indicates adifference between the left and right eye images. If the differenceindicator value indicates a non-zero difference, the determinationmodule 1432 determines that entropy reduction operation was performedone of the left and right eye images.

The sharpening filter module 1436 performs a sharpening filteringoperation using decoded eye image data corresponding to the one of theleft and right eye images which was subject to an entropy reductionoperation, to produce a sharpened version of the entropy reduced imagewhen it is determined that that one of the left and right eye images wassubject to an entropy reduction operation prior to encoding, the entropyreduced image being the one of the left and right eye images which wassubject to an entropy reduction operation. The sharpening operation isperformed, e.g., to increase visual edge prominence within entropyreduced eye image. The sharpening filter control module 1438 controlsthe strength of the sharpening filter 1436 used to perform thesharpening operation as a function of the difference indicator valuewhen it is determined that that one of said left and right eye imageswas subject to an entropy reduction operation, the control module 1438controlling a higher amount of sharpening to be performed when thedifference indicator value indicates a smaller difference than when thedifference indicator value indicates a larger difference.

The edge enhancement module 1440 performs edge enhancement operation onthe entropy reduced eye image regions, e.g., image regions of one of theleft and right eye image which was subject to an entropy reductionoperation. In various embodiments the entropy reduced eye image regionsare defined by a mask created using the polygonal region information,e.g., image portions identified as negative parallax regions, byinformation associated with the entropy reduced image as discussedearlier. Thus the edge enhancement module 1440 performs sharpening ofthe decoded entropy reduced eye image. This sharpening may compensatefor softened image edges in the entropy reduction process that mightdestructively interfere with stereoscopic depth cues.

The rendering module 1442 performs rendering operation rendering a 3-Dimage from the sharpened version of the entropy reduced image (orunsharpened version depending on whether entropy reduction waspreviously applied or not) and the decoded other one of the left andright eye images which was not subject to an entropy reductionoperation. A rendered 3-D image 1450 may be then displayed on thedisplay 1402.

In accordance with some embodiments an exemplary method of operating anencoder apparatus to encode stereoscopic image data representing pairsof first and second eye images, each first eye image in an image paircorresponding to one of a left eye image and a right eye image, thesecond eye image of each image pair corresponding to the other one ofthe left and right eye images, comprises: determining an amount ofdifference between a first eye image and a second eye image of a firstimage pair; when it is determined that there is a non-zero amount ofdifference between the first eye image and the second eye image of saidfirst image pair, performing the steps of: i) performing an entropyreduction operation on second eye image data representing the second eyeimage of the first image pair to produce entropy reduced second eyeimage data; ii) encoding first eye image data representing the first oneof the eye images of said first image pair to generate compressed firsteye image data; and iii) encoding said entropy reduced second eye imagedata to generate compressed second eye image data; and storing ortransmitting the compressed first eye image data and compressed secondeye image data.

In some embodiments said encoding said entropy reduced second eye imagedata includes supplying said entropy reduced second eye image data to anencoder which performs inter-frame motion compensated encoding, datacorresponding to said first and second eye images being processed asdata corresponding to sequential frames. In some embodiments saidentropy reduction operation is a function of the determined amount ofdifference between said first eye image and said second eye image, theamount of entropy reduction being greater when the amount of differenceis a first amount than when it is a second amount, said second amountbeing greater than said first amount.

In some embodiments said entropy reduction operation includes performingat least one of an FFT based filtering operation or a neighborhoodfiltering operation on said second eye image data to smooth the secondeye image of said first image pair. In some embodiments the exemplaryencoding method further includes: performing a negative parallaxdetection operation to detect image regions in said second eye image ofsaid first image pair which are in negative parallax relative to acorresponding portion of said first eye image of said first image pair;and storing information identifying at least one detected image regionof negative parallax in said second eye image of said first image pairin a file with said compressed first eye image data and compressedsecond eye image data.

In some embodiments the exemplary encoding method further includes:storing information indicating said amount of difference in said filewith said compressed first eye image data and compressed second eyeimage data. In some embodiments the detected image regions are polygonalimage regions; and the information identifying at least one detectedimage region of negative parallax includes information defining theperimeter of at least one polygonal image region.

In some embodiments the information defining the perimeter of the atleast one polygonal image region of negative parallax includesinformation identifying points in said second eye image corresponding tothe perimeter and a formula for determining a portion of said perimeterfrom said identified locations of points corresponding to the perimeter.

In some embodiments when it is determined that there is a zero amount ofdifference between the first eye image and the second eye image of saidfirst image pair, the exemplary method of encoding comprises performing,prior to performing said storing or transmitting step, the steps of: ii)encoding said first eye image data to generate compressed first eyeimage data; and iii) encoding said second eye image data to generatecompressed second eye image data. In some embodiments encoding saidsecond eye image data generates compressed second eye image dataindicating that the second eye image is the same as the first eye image.In various embodiments the exemplary encoding method further includes:processing a second image pair including a left eye image and a righteye image.

In accordance with some embodiments an exemplary apparatus to encodestereoscopic image data representing pairs of first and second eyeimages, comprises: a determination module for determining an amount ofdifference between a first eye image and a second eye image of a firstimage pair; an entropy reduction filtering module for performing anentropy reduction operation on second eye image data representing thesecond eye image of the first image pair to produce entropy reducedsecond eye image data, when it is determined that there is a non-zeroamount of difference between the first eye image and the second eyeimage of said first image pair; a compression module for encoding firsteye image data representing the first one of the eye images of saidfirst image pair to generate compressed first eye image data, and forencoding said entropy reduced second eye image data to generatecompressed second eye image data; and a storage module for storing thecompressed first eye image data and compressed second eye image data. Insome embodiments the exemplary apparatus further includes: an interfacefor transmitting the compressed first eye image data and compressedsecond eye image data;

In some embodiments the compression module for encoding said entropyreduced second eye image data includes an encoder which performsinter-frame motion compensated encoding, data corresponding to saidfirst and second eye images being processed as data corresponding tosequential frames. In some embodiments the entropy reduction operationis a function of the determined amount of difference between said firsteye image and said second eye image, the amount of entropy reductionbeing greater when the amount of difference is a first amount than whenit is a second amount, said second amount being greater than said firstamount.

In some embodiments the entropy reduction filtering module includes amodule for performing at least one of an FFT based filtering operationor a neighborhood filtering operation on said second eye image data tosmooth the second eye image of said first image pair. In someembodiments the exemplary apparatus for encoding further comprises: anegative parallax detection module for performing a negative parallaxdetection operation to detect image regions in said second eye image ofsaid first image pair which are in negative parallax relative to acorresponding portion of said first eye image of said first image pair.In some embodiments the storage module further stores informationidentifying at least one detected image region of negative parallax insaid second eye image of said first image pair in a file with saidcompressed first eye image data and compressed second eye image data.

In some embodiments the storage module further stores informationindicating said amount of difference in said file with said compressedfirst eye image data and compressed second eye image data.

In some embodiments the detected image regions are polygonal imageregions; and the information identifying at least one detected imageregion of negative parallax includes information defining the perimeterof at least one polygonal image region. In some embodiments theinformation defining the perimeter of the at least one polygonal imageregion of negative parallax includes information identifying points insaid second eye image corresponding to the perimeter and a formula fordetermining a portion of said perimeter from said identified locationsof points corresponding to the perimeter.

In accordance with one embodiment an exemplary method of processingencoded frames of stereoscopic image data, comprises: decoding first andsecond encoded frames representing first and second eye images of animage pair to recover decoded first and second eye image data;determining if one of said first and second eye images was subject to anentropy reduction operation prior to encoding; when it is determinedthat that one of said first and second eye images was subject to anentropy reduction operation prior to encoding: i) performing asharpening operation, using decoded eye image data corresponding to theone of the first and second eye images which was subject to an entropyreduction operation, to produce a sharpened version of the entropyreduced image, said entropy reduced image being the one of the first andsecond eye images which was subject to an entropy reduction operation;and ii) rendering a 3-D image from said sharpened version of the entropyreduced image and the decoded one of the first and second eye imageswhich was not subject to an entropy reduction operation.

In accordance with one feature of some embodiments the sharpeningoperation increases a visual edge prominence within the image beingsubject to said sharpening. In some embodiments determining if one ofsaid first and second eye images was subject to an entropy reductionoperation prior to encoding includes determining if there is adifference between said first and second eye images. In some embodimentsdetermining if one of said first and second eye images was subject to anentropy reduction operation prior to encoding includes determining if adifference indicator value associated with at least one of said firstand second encoded frames indicates a difference between the first andsecond eye images.

In some embodiments when it is determined that that one of said firstand second eye images was subject to an entropy reduction operation, theexemplary method processing encoded frames of stereoscopic image datafurther comprises: controlling the strength of a sharpening filter usedto perform said sharpening operation as a function of said differenceindicator value, a higher amount of sharpening being performed when saiddifference indicator value indicates a smaller difference than when saiddifference indicator value indicates a larger difference. In someembodiments when it is determined that that one of said first and secondeye images was subject to an entropy reduction operation prior toencoding, the exemplary method includes performing an edge enhancementoperation on said sharpened version of the entropy reduced image priorto using said sharpened version of the entropy reduced image in saidrendering step.

In some embodiments the edge enhancement operation is performed on imageportions identified as negative parallax regions, by informationassociated with said entropy reduced image. In some embodiments theexemplary method of processing encoded frames of stereoscopic image dataincludes: receiving a file including said encoded frames of stereoscopicimage data, said file including a plurality of frame pairs which wereinterleaved to form a left and right eye image sequence prior toencoding. In some embodiments information is included in said receivedfile for at least one pair of frames indicating an amount of differencebetween the images represented by said at least one pair of frames andinformation identifying at least one region of negative parallax in oneof said frames which was generated from an image subjected to an entropyreduction operation.

In some embodiments when said determining indicates that one of saidfirst and second eye images was not subject to an entropy reductionoperation, the method further includes rendering a 3-D image from saidfirst and second eye image data without performing a sharpeningoperation.

In accordance with one embodiment, an exemplary apparatus for processingencoded frames of stereoscopic image data, comprises: a decoder modulefor decoding first and second encoded frames representing first andsecond eye images of an image pair to recover decoded first and secondeye image data; an entropy reduction determination module fordetermining if one of said first and second eye images was subject to anentropy reduction operation prior to encoding; a sharpening filtermodule for performing a sharpening operation, using decoded eye imagedata corresponding to the one of the first and second eye images whichwas subject to an entropy reduction operation, to produce a sharpenedversion of the entropy reduced image when it is determined that that oneof said first and second eye images was subject to an entropy reductionoperation prior to encoding, said entropy reduced image being the one ofthe first and second eye images which was subject to an entropyreduction operation; and a rendering module for rendering a 3-D imagefrom said sharpened version of the entropy reduced image and the decodedone of the first and second eye images which was not subject to anentropy reduction operation.

In some embodiments the exemplary apparatus further includes aninterface for receiving a file including said encoded frames ofstereoscopic image data, said file including a plurality of frame pairswhich were interleaved to form a left and right eye image sequence priorto encoding.

In some embodiments said sharpening operation performed by thesharpening filter module increases visual edge prominence within theimage being subject to said sharpening. In some embodiments the entropyreduction determination module includes a difference determinationmodule for determining if there is a difference between said first andsecond eye images. In some embodiments the entropy reductiondetermination module is further configured to determine if a differenceindicator value associated with at least one of said first and secondencoded frames indicates a difference between the first and second eyeimages.

In some embodiments the exemplary apparatus further comprises a controlmodule for controlling the strength of a sharpening filter used toperform said sharpening operation as a function of said differenceindicator value when it is determined that that one of said first andsecond eye images was subject to an entropy reduction operation, saidcontrol module controlling a higher amount of sharpening to be performedwhen said difference indicator value indicates a smaller difference thanwhen said difference indicator value indicates a larger difference. Insome embodiments the exemplary apparatus further includes an edgeenhancement operation module for performing an edge enhancementoperation on said sharpened version of the entropy reduced image. Insome embodiments the edge enhancement operation is performed on imageportions identified as negative parallax regions, by informationassociated with said entropy reduced image.

Some advantages of the present invention which should be appreciatedinclude, without limitation, that stereoscopic video can be distributedto customers using less storage and transmission bandwidth thanconventional stereoscopic video and, furthermore, that the player'sprocessor, e.g., graphic processing unit, can be applied to reduce thehardware burden of rendering stereoscopic video to the display.

Some embodiments are directed a non-transitory computer readable mediumembodying a set of software instructions, e.g., computer executableinstructions, for controlling a computer or other device to encode andcompresses stereoscopic video. Other embodiments are embodiments aredirected a computer readable medium embodying a set of softwareinstructions, e.g., computer executable instructions, for controlling acomputer or other device to decode and decompresses video on the playerend. While encoding and compression are mentioned as possible separateoperations, it should be appreciated that encoding may be used toperform compression and thus encoding may, in some include compression.Similarly, decoding may involve decompression.

The techniques of various embodiments may be implemented using software,hardware and/or a combination of software and hardware. Variousembodiments are directed to apparatus, e.g., a video data processingsystem. Various embodiments are also directed to methods, e.g., a methodof processing video data. Various embodiments are also directed tomachine, e.g., computer, readable medium, e.g., ROM, RAM, CDs, harddiscs, etc., which include machine readable instructions for controllinga machine to implement one or more steps of a method.

Various features of the present invention are implemented using modules.Such modules may, and in some embodiments are, implemented as softwaremodules. In other embodiments the modules are implemented in hardware.In still other embodiments the modules are implemented using acombination of software and hardware. A wide variety of embodiments arecontemplated including some embodiments where different modules areimplemented differently, e.g., some in hardware, some in software, andsome using a combination of hardware and software. It should also benoted that routines and/or subroutines, or some of the steps performedby such routines, may be implemented in dedicated hardware as opposed tosoftware executed on a general purpose processor. Such embodimentsremain within the scope of the present invention. Many of the abovedescribed methods or method steps can be implemented using machineexecutable instructions, such as software, included in a machinereadable medium such as a memory device, e.g., RAM, floppy disk, etc. tocontrol a machine, e.g., general purpose computer with or withoutadditional hardware, to implement all or portions of the above describedmethods. Accordingly, among other things, the present invention isdirected to a machine-readable medium including machine executableinstructions for causing a machine, e.g., processor and associatedhardware, to perform one or more of the steps of the above-describedmethod(s).

Numerous additional variations on the methods and apparatus of thevarious embodiments described above will be apparent to those skilled inthe art in view of the above description. Such variations are to beconsidered within the scope.

What is claimed is:
 1. A method of operating an encoder apparatus toencode stereoscopic image data representing pairs of first and secondeye images, each first eye image in an image pair corresponding to oneof a left eye image and a right eye image, the second eye image of eachimage pair corresponding to the other one of the left and right eyeimages, the method comprising: determining an amount of differencebetween a first eye image and a second eye image of a first image pair;when it is determined that there is a non-zero amount of differencebetween the first eye image and the second eye image of said first imagepair, performing the steps of: i) performing an entropy reductionoperation on second eye image data representing the second eye image ofthe first image pair to produce entropy reduced second eye image data;ii) encoding first eye image data representing the first one of the eyeimages of said first image pair to generate compressed first eye imagedata; and iii) encoding said entropy reduced second eye image data togenerate compressed second eye image data; and storing or transmittingthe compressed first eye image data and compressed second eye imagedata.
 2. The method of claim 1, wherein said encoding said entropyreduced second eye image data includes supplying said entropy reducedsecond eye image data to an encoder which performs inter-frame motioncompensated encoding, data corresponding to said first and second eyeimages being processed as data corresponding to sequential frames. 3.The method of claim 1, wherein said entropy reduction operation is afunction of the determined amount of difference between said first eyeimage and said second eye image, the amount of entropy reduction beinggreater when the amount of difference is a first amount than when it isa second amount, said second amount being greater than said firstamount.
 4. The method of claim 1, wherein said entropy reductionoperation includes performing at least one of an FFT based filteringoperation or a neighborhood filtering operation on said second eye imagedata to smooth the second eye image of said first image pair.
 5. Themethod of claim 1, further comprising: storing information indicatingsaid amount of difference in said file with said compressed first eyeimage data and compressed second eye image data.
 6. The method of claim1, further comprising: when it is determined that there is a zero amountof difference between the first eye image and the second eye image ofsaid first image pair performing, prior to performing said storing ortransmitting step, the steps of: ii) encoding said first eye image datato generate compressed first eye image data; and iii) encoding saidsecond eye image data to generate compressed second eye image data. 7.The method of claim 6, wherein said encoding said second eye image datagenerates compressed second eye image data indicating that the secondeye image is the same as the first eye image.
 8. The method of claim 1,further comprising: processing a second image pair including a left eyeimage and a right eye image.
 9. An apparatus to encode stereoscopicimage data representing pairs of first and second eye images, each firsteye image in an image pair corresponding to one of a left eye image anda right eye image, the second eye image of each image pair correspondingto the other one of the left and right eye images, the apparatuscomprising: a determination module for determining an amount ofdifference between a first eye image and a second eye image of a firstimage pair; an entropy reduction filtering module for performing anentropy reduction operation on second eye image data representing thesecond eye image of the first image pair to produce entropy reducedsecond eye image data, when it is determined that there is a non-zeroamount of difference between the first eye image and the second eyeimage of said first image pair; a compression module for encoding firsteye image data representing the first one of the eye images of saidfirst image pair to generate compressed first eye image data, and forencoding said entropy reduced second eye image data to generatecompressed second eye image data; and a storage module for storing thecompressed first eye image data and compressed second eye image data.10. The apparatus of claim 9, further comprising: an interface fortransmitting the compressed first eye image data and compressed secondeye image data.
 11. The apparatus of claim 12, wherein said compressionmodule for encoding said entropy reduced second eye image data includesan encoder which performs inter-frame motion compensated encoding, datacorresponding to said first and second eye images being processed asdata corresponding to sequential frames.
 12. The apparatus of claim 9,wherein said entropy reduction operation is a function of the determinedamount of difference between said first eye image and said second eyeimage, the amount of entropy reduction being greater when the amount ofdifference is a first amount than when it is a second amount, saidsecond amount being greater than said first amount.
 13. The apparatus ofclaim 9, wherein said entropy reduction filtering module includes amodule for performing at least one of an FFT based filtering operationor a neighborhood filtering operation on said second eye image data tosmooth the second eye image of said first image pair.
 14. The apparatusof claim 9, wherein said storage module further stores informationindicating said amount of difference in said file with said compressedfirst eye image data and compressed second eye image data.